Designing for gestural interaction in VR By a Design Director
Matt Corrall is currently the Design Director of Ultraleap, where he is responsible for inventing controller-free haptics and hand-tracking. Matt has led the team in establishing some of the earliest standards and published guidelines for gestural interfaces, drawing on his broad experience in industrial product and user experience design. These were some of the initial standards.
Recent conversations I've had with Matt have always been really interesting. We talked about his beginnings in virtual reality, the fascinating projects being worked on at Ultraleap, as well as his approach to designing for hand-tracking and haptics. Several important passages from Matt's observations are included in the following quotations.
Journey into a VR career
My prior experience is in the design of industrial products. Over the course of several years, I made a slow but steady transition into digital and user experience design. In addition to that, I've had a lot of experience working with 3D Modeling services design and user research.
My first experience with a business named DNV, which specializes in renewable energy, sparked my interest in the field of virtual reality. We were developing enterprise software for designing wind farms and solar farms, as well as other applications like performing safety inspections on large cargo ships. A one-off prototype was developed by our team so that safety checks on ships may be performed remotely using virtual reality. During the course of working on this project, I came to the conclusion that traditional design principles cannot be applied to virtual reality.
As a small group consisting of only three people, we were learning fundamental concepts as we went along. It was motivating to see that. I had always intended to return to virtual reality thereafter, which is what brought me to Ultraleap.
Life as a design director
I came to Ultraleap with the goal of establishing a user research and design team and introducing user-centered ways of thinking and doing work to the company, which has traditionally been focused on engineering. Working as a designer is less strategically demanding than working as a director. I work hard to put in place effective procedures and improve everyone's skills. I assist teams in comprehending how to follow proper user tested design and how to carry out research. Our user research even contributes to adjusting Ultraleap's strategy and research and development efforts.
I still roll my sleeves up and get dug in! On the other hand, the majority of my time is spent on management. That's what I really appreciate, since it's the thing that sets this place apart. We began as a little start-up with a concentration on hackathons and quick ideas while we were on the move. We have evolved into an evidence-based scale-up that engages in iterative improvement and places a primary emphasis on our clients.
Ultraleap hand-tracking & haptics
Hand-tracking
Ultraleap is the creator of the most cutting-edge hand-tracking technology in the world. It perfectly replicates the nuanced complexities and subtleties of natural hand movements. A decade's worth of work and iteration, generations' worth of research in artificial intelligence, and the input of thousands of developers went into its creation.
Contactless haptics
The haptics technology developed by Ultraleap replicates the feeling of touch, but it does so in the air by employing a number of extremely small ultrasonic speakers. Your palm and fingers will feel a vibration that is pinpoint accurate but not painful as a result of the ultrasonic waves.
When combined with hand tracking, this gives you the ability to, for example, touch a virtual object in VR while simultaneously feeling the sensation on your fingertips.
Standout projects at Ultraleap
Lufthansa
Our hand-tracking technology is used by several of our customers in the development of training applications. They are able to get the most out of hand-tracking thanks to the direct support we provide as well as the instructions we provide.
A good illustration of this would be the Lufthansa airline. They have a virtual reality (VR) app that can be used to teach flight attendants. Cabin crew members have a short amount of time after a trip to clean the aircraft and do any necessary safety inspections once they land. Making sure that everything is where it should be, such as verifying that the cabin lockers are closed, etc. Because training in a real plane takes a lot of time and is expensive, VR is a cost-effective alternative.
There were some people who were trying virtual reality for the very first time. In the future, they will most likely only employ virtual reality for training purposes once per twelve months. Therefore, it was vital for our client to shorten the learning curve and cut down on extraneous distractions. There is not much time available for someone to practice what they will need to know for the interactions (e.g. pinch gestures). People have a propensity to respond and grab objects in the same way that they would in real life.
Testing allowed us to get insight into the various ways in which individuals respond to'real-world' interfaces. We had to take into account the fact that people grab and hold things in a variety of different ways. Therefore, objects like handles and buttons needed to have numerous ways of being activated so that they could accommodate to the various actions. In addition, the introduction of any new idea, such as a hand position, was done in stages.
When designing the user interface, we favored the use of real-world items whenever possible. Instead of using popups, buttons, or drop-down menus, we decided to employ real things and simulate the laws of physics.
Approach to a VR project
Research on end-users
We research various products and interactions available on the market, in addition to speaking with users and consumers. We conduct research into the various applications to see which ones would benefit most from hand-tracking and then base our designs on those findings. Therefore, it was more comfortable for the user to use their hands and known interactions rather than controllers in the case of Lufthansa. This would result in a quicker on-boarding process.
Planning
We want to know what kinds of relationships are necessary. What sort of gestures? How do customers navigate their way through the space? We create a map for each possible use case. After then, our ideas for these use cases begin to diverge.
For instance, at the moment we are investigating ways to enable players to teleport around a level without the usage of a controller. Users are typically instructed to keep a button depressed in order to select a new location to teleport to. Without an established standard of excellence, this may be a characteristic of the highest priority that is also essential to the overall experience. These use cases are prioritised, and design sprints are planned for them, so that we can design for the various interactions and prototype for testing.
Design
Iterating on concepts while working directly with the engineers helps us. First, we create sketches, and then we bring those concepts to life in 3D. In order to locate the most effective prototyping tool for this, we have utilised a variety of options and are continuing our search. We go through concepts at a very rapid rate.
As we go closer together, we will begin constructing things using either Unity or Unreal (mostly Unity, as more developers are trained in this).
Testing
After a few days, interactions are typically in a condition that is satisfactory enough for testing with users. During these meetings, we put our hypotheses to the test and discuss how to refine our ideas. We perform additional iterations on designs that are selected to advance to the next level.
In the case of teleportation, for instance, we put five different concepts through user testing. We got rid of two and focused on improving the performance of the remaining three. After that, we proceeded to do additional tests.
As testing progresses, we put interactions through their paces in a variety of application environments to see how best to tailor them to specific use cases. Examples include virtual reality arcade games and flying simulators.
Establish the connections between them
When we are satisfied that a given interaction is discoverable, useable, technically sound, and works effectively in conjunction with other interactions, we add it to our software development kit (SDK) and design guidelines so that customers can use it. We make it a point to get their comments on how they've implemented it in their own projects as well as suggestions for how we can improve what we offer.
We might also put these ideas through more testing at conferences and demonstrations. For instance, we are prepared to present demos at CES the following year.
